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The  objective  of  this  project  is  to  train  graduate  students  in  conducting  research 
in  human  factors  and  ergonomics  issues  related  to  aviation.  Numerous  research 
projects  have  been  completed  within  this  project.  They  have  focused  on:  (1)  an 
investigation  of  causal  factors  related  to  problems  with  sutuation  awareness  among  | 
pilots  and  air  traffic  controllers,  (2)  investigatins  of  automated  compensation  ^ 

for  enhancing  performance  in  teleoperations  task  under  operator/system  control  axis  l 
misalignment  with  both  static  and  dynamic  misalignments,  (3)  an  investigation  of 
level  of  automation  effects  on  human  performance,  situation  awareness  and  workload,  : 
(4)  an  investigation  of  adaptive  automation  and  level  of  control  on  human  perfor¬ 
mance,  situation  awareness  and  workload  under  dual  task  conditions,  and  (5)  an 
investigation  of  cue  characteristics  related  to  errors  in  situation  awareness  con-  | 
prehension.  Each  of  these  projects  provided  the  AASERT  fellows  with  extensive  ^ 

experience  in  studying  critical  issues  related  to  human  performance  in  aviation,  ^ 
and  each  has  resulted  in  new  research  that  significantly  advances  scientific 
knowledge  in  each  of  these  areas.  ^ 
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Objective 


The  objective  of  this  project  is  to  train  graduate  students  in  conducting  research 
in  human  factors  and  ergonomics  issues  related  to  aviation. 

Status  of  Effort 

Two  students  were  supported  under  this  grant. 

David  B.  Kaber:  Graduated  with  a  Ph.D  in  Industrial  Engineering  -  August, 
1996.  GPA  3.95/4.0 

Dissertation:  The  effect  of  level  of  automation  and  adaptive  automation 
on  performance  in  dynamic  control  environments 

Debra  G.  Jones:  Graduated  with  a  Ph.D.  in  Industrial  Engineering  - 
December,  1996.  GPA  3.96/4.0 

Dissertation:  The  effect  of  cue  type  on  situation  awareness 

Research  Conducted  by  AFOSR  Student  Fellows 

Numerous  research  projects  have  been  completed  within  this  project.  They 
have  focused  on:  (1)  an  investigation  of  causal  factors  related  to  problems  with 
situation  awareness  among  pilots  and  air  traffic  controllers,  (2)  investigations  of 
automated  compensation  for  enhancing  performance  in  teleoperations  task 
under  operator/system  control  axis  misalignment  with  both  static  and  dynamic 
misalignments,  (3)  an  investigation  of  level  of  automation  effects  on  human 
performance,  situation  awareness  and  workload,  (4)  an  investigation  of 
adaptive  automation  and  level  of  control  on  human  performance,  situation 
awareness  and  workload  under  dual  task  conditions,  and  (5)  an  investigation  of 
cue  characteristics  related  to  errors  in  situation  awareness  comprehension. 
Each  of  these  projects  provided  the  AASERT  fellows  with  extensive  experience 
in  studying  critical  issues  related  to  human  performance  in  aviation,  and  each 
has  resulted  in  new  research  that  significantly  advances  scientific  knowledge  in 
each  of  these  areas. 


Accomplishments/Findinqs 

The  following  studies  were  completed  under  this  grant. 

(1)  Investigation  of  causal  factors  related  to  problems  with  situation  awareness 
among  pilots  and  air  traffic  controllers 

Situation  Awareness  (SA)  is  a  crucial  factor  in  effective  decision  making  in  the 
dynamic  flight  environment.  Consequently,  an  understanding  of  the  types  of  SA 
errors  that  occur  in  this  environment  is  beneficial.  This  study  used  reports  from 
the  Aviation  Safety  Reporting  System  (ASRS)  to  investigate  the  types  of  SA 
errors  that  occur  in  aviation  and  causal  factors  associated  with  their  occurrence. 


The  errors  were  classified  into  one  of  three  major  categories:  Level  1  (failure  to 
correctly  perceive  the  information),  Level  2  (failure  to  comprehend  the  situation), 
or  Level  3  (failure  to  project  the  situation  into  the  future).  Of  the  errors  identified, 
78%  were  Level  1  SA  errors,  17%  were  Level  2  SA  errors,  and  5%  were  Level 
3  SA  errors.  Level  1  SA  errors  occurred  when  relevant  data  was  not  available, 
when  data  was  hard  to  discriminate  or  detect,  when  a  failure  to  monitor  or 
observe  data  occurred,  when  presented  information  was  misperceived,  or  when 
memory  loss  occurred.  Level  2  SA  errors  involved  a  missing/or  incomplete 
mental  model,  the  use  of  an  incorrect  mental  model,  over-reliance  on  default 
values,  and  miscellaneous  other  factors.  Level  3  errors  involved  either  an  over¬ 
projection  of  current  trends  or  miscellaneous  other  factors.  These  results 
provided  indication  of  the  types  and  frequency  of  SA  errors  that  occur  in 
aviation  and  an  indication  of  factors  that  lead  to  problems  in  SA  along  with  their 
respective  frequencies. 

This  research  provides  significant  new  insight  into  the  types  of  problems  pilots 
have  with  situation  awareness,  and  the  types  of  factors  that  lead  to  SA  errors. 

As  situation  awareness  has  been  found  to  be  a  causal  factor  in  88%  of  aircraft 
accidents,  these  results  are  particularly  important  to  the  goal  of  improving  safety 
and  decreasing  accidents  in  aircraft  both  in  military  and  civilian  aviation.  The 
data  will  be  used  to  determine  remedial  techniques  for  improving  SA  within 
each  causal  factor  category. 

(2)  Investigations  of  automated  compensation  for  enhancing  performance  in 
teleoperations  task  under  ooerator/system  control  axis  misalignment  with  both 
static  and  dynamic  misalignments 

The  first  study  examined  the  effect  of  automatic  compensation  for  misalignment 
between  display  and  controller  orientation  on  teleoperation.  Teleoperation 
tasks  are  common  in  aviation  for  activities  such  as  satellite  maneuvering  in 
space  or  the  operation  of  aircraft  drones  for  example.  In  a  telemanipulation 
scenario  gross  misalignments  may  arise  between  operator  viewing  direction 
and  controller  orientation  due  to  changing  system  (end-effector)  or  operator 
positions.  It  remains  to  be  clarified  as  to  whether  humans  can  adapt  to  such 
visual-motor  non-compliance  through  experience.  An  experiment  was 
conducted  to  test  a  means  of  providing  automated  compensation  for 
misalignments  between  display  and  controller  rotations  relieving  human 
operators  of  the  need  to  learn  or  adjust  for  alternate  motor  mappings.  Subjects 
were  asked  to  perform  a  telemanipulation  task  involving  tracking  of  a  randomly 
moving  target.  Performance  was  recorded  under  various  misalignments  of  the 
controller  (from  0  to  180  degrees  in  each  direction)  with  and  without  automated 
compensation.  Results  indicate  significant  improvements  in  operator 
performance  through  use  of  automatic  compensation  under  misalignments 
involving  both  the  controller  and  display  (such  as  encountered  in  situations 
where  the  human  operator  moves  about  a  fixed  teleoperator).  When  only  the 
controller  is  misaligned  (such  as  encountered  in  situations  where  the 
teleoperator  moves  about  a  fixed  human  operator  and  the  human  moves  his  or 
her  head  to  maintain  display  orientation),  human  operators  are  able  to  adjust 
internally,  and  the  automated  compensation  actually  degraded  performance. 


This  study  was  followed  by  a  second  study  which  replicated  the  first,  but  was 
expanded  to  examine  the  effect  of  dynamically  changing  orientations  of  the 
human  operator  or  the  display  as  opposed  to  the  fixed  misalignments  examined 
in  the  first  study.  Teleoperations  are  often  performed  under  such  positional 
disturbances  which  may  be  brought  on  by  movement  of  the  operator  or 
teleoperator.  An  experiment  was  conducted  in  which  subjects  were  required  to 
perform  teleoperation  involving  tracking  an  irregularly  moving  target  on  a 
computer  display  using  a  cursor  linked  to  a  joystick  while  they,  or  the  display, 
were  continuously  rotated  in  a  clockwise  or  counter-clockwise  direction. 
Performance  was  recorded  under  different  rates  of  rotation,  with  and  without 
automated  compass  compensation  for  the  misalignment.  Results  indicate  that 
significant  improvements  in  operator  task  accuracy  were  achieved  with  the 
compass  for  situations  in  which  the  teleoperator  was  rotated  (leading  to 
misalignments  of  the  display  and  controller),  however  not  in  the  situation  in 
which  the  misalignments  were  brought  on  by  the  rotation  of  the  human  operator 
(leading  to  misalignments  of  the  controller  alone).  Thus,  the  second  study 
supported  the  findings  of  the  first  study  under  dynamic  misalignment  conditions. 

These  two  studies  provide  significant  new  data  providing  an  understanding  of 
what  misalignment  conditions  cause  problems  for  human  operators  in 
teleoperations  tasks  and  showing  considerable  benefit  to  the  use  of  automated 
compensation  under  some  conditions.  These  results  are  applicable  to  a  range 
of  situations  encountered  by  the  Air  Force  in  which  human  operators  rnay 
control  dynamic  objects  that  are  separate  from  them  (e.g.  drones,  mobile  robots, 
etc...)  or  in  which  they  need  to  operate  the  dynamic  control  of  objects  while 
moving  themselves  (e.g.  operating  on  satellites  in  space). 

(3)  Investigation  of  level  of  automation  effects  on  human  performance,  situation 
awareness  and  workload. 

Automation  has  become  wide  spread  in  aviation,  both  in  the  cockpit  and  the 
ground.  Accompanying  the  use  of  automation,  a  noted  trend  towards  out-of-the- 
loop  performance  errors  has  occurred.  Historically,  the  pilot  has  been  expected 
to  maintain  a  monitoring  role  over  automation,  intervening  when  necessary. 
Their  ability  to  do  so  when  functioning  in  such  a  role  is  quite  limited,  however. 

An  out-of-the-loop  performance  problem  results  in  which  humans  have  a 
significant  reduction  in  their  ability  to  detect  system  errors  and  perform  tasks 
manually  in  the  face  of  system  failures.  This  problem  has  been  shown  to  be 
largely  due  to  a  loss  of  situation  awareness  that  occurs  under  automated 
conditions.  Furthermore,  such  systems  tend  to  be  very  difficult  for  human 
operators  to  control  and  understand  in  real-time  due  to  the  increased  level  of 
complexity  associated  with  most  automated  systems. 

This  project  investigated  the  use  of  intermediate  levels  of  automation  in  which 
the  human  operator  maintains  a  greater  degree  of  control  as  a  means  of 
improving  situation  awareness  and  avoiding  the  out-of-the  loop  syndrome 
associated  with  full  automation  of  tasks.  A  ten  level  taxonomy  of  Level  of 
Automtion  (LOA)  was  developed  that  delineates  different  levels  of  automation 


as  a  function  of  which  roles  are  assigned  to  the  human  operator  and  which  are 
assigned  to  the  machine.  The  taxonomy  was  developed  to  incorporate 
functions  that  needed  to  be  performed  across  a  wide  range  of  systems, 
including  teleoperations  and  robotics.  The  four  major  role  types  were  defined 
as:  Monitoring  (sensing  and  observing  information),  Generating  (producing 
options  for  control  performance).  Selection  (choosing  which  option  to 
implement)  and  Implementing  (actually  performing  the  task).  The  ten  levels  of 
automaton  were  formed  on  the  basis  of  allocating  each  role  to  the  human, 
computer  or  shared  between  the  human  and  computer.  They  include  (1) 

Manual  control,  (2)  Action  support,  (3)  Batch  processing,  (4)  Shared  control,  (5) 
Decision  support,  (6)  Blended  decision  making,  (7)  Rigid  system,  (8)  Automated 
decision  making,  (9)  Supervisory  control ,  and  (10)  Full  automation. 

A  study  was  conducted  using  a  simulation  that  incorporated  multiple  tasks  to  be 
processed  with  different  relevance  to  the  assigned  goals,  multiple  goals,  and 
high  task  demands  under  limited  time  resources.  The  simulation  required  users 
to  develop  complex  strategies  for  optimizing  system  performance  along  multiple 
goals  simultaneously.  The  study  revealed  that  overall  performance  was  greatly 
improved  by  LOA  that  provided  computer  aiding  in  the  Implementation  role  of 
task  execution.  Decrements  in  overall  performance  were  obsen/ed,  however, 
with  LOA  that  incorporated  joint  human-computer  Generation  of  options,  as 
compared  to  purely  manual  Generation.  The  use  of  computer  or  joint  human- 
computer  Selection  of  options  did  not  significantly  affect  overall  performance, 
compared  to  human  Selection.  (Joint  Monitoring  was  necessary  across  almost 
all  levels  of  control).  When  the  automation  failed,  and  humaR-operators  were 
required  to  jump-in  and  assume  manual  control,  their  ability  to  do  so  was 
significantly  better  when  human  functioning  in  the  Implementation  role  had 
been  present.  Time  to  recover  and  performance  during  manual  control 
immediately  following  the  failure  was  worse  at  LOA  that  incorporated  advanced 
process  planning  in  the  queuing  of  targets  (Batch  Processing  and  Automated 
Decision  Making).  Reductions  in  workload  and  corresponding  improvements  in 
operator  situation  awareness  were  observed  with  higher  LOA  that  incorporated 
joint  human-  computer  or  computer  Selection  of  options  to  implement. 

This  study  represents  a  unique  contribution  to  the  study  of  automation  in 
aviation,  both  in  terms  of  the  LOA  taxonomy  developed  and  in  terms  of  the 
findings  of  the  study  regarding  which  aspects  of  a  task  benefit  most  from 
automation  (or  may  be  degraded  by  automation).  As  automation  is  becoming 
an  increasing  part  of  many  aviation  systems  (including  commercial  and  military 
cockpits,  air  traffic  control  and  ground  control),  expanding  from  simply 
performing  tasks  or  control  functions  to  providing  decision  guidance  and  highly 
cognitive  functions  (through  artificial  intelligence  and  expert  systems,  for 
instance),  these  findings  are  particularly  germane.  These  results  are  directly 
applicable  to  design  decisions  regarding  which  tasks  automation  should 
perform  and  how  such  systems  should  interact  with  human  operators. 


(4)  Investigation  of  adaptive  automation  and  level  of  automation  on  human 
performance,  situation  awareness  and  workload  under  dual  task  conditions. 

This  study  expanded  on  the  previous  study  investigating  the  effects  of  level  of 
automation  (LOA)  to  include  an  investigation  of  adaptive  automation  (AA)  and 
dual-task  conditions.  Most  automation  does  not  function  all  the  time,  but  rather 
at  different  times  control  may  be  passed  to  human  operators  for  particular  tasks. 
Intermittent  periods  of  transfer  of  control  may  serve  to  improve  human 
performance  by  keeping  the  operator  in-the-loop.  Effective  strategies  for 
merging  control  from  multiple  sources  need  to  be  developed  and  for 
transitioning  control  from  machines  to  humans. 

This  study  examined  several  AA  partitioning  strategies  in  which  the  percentage 
of  the  time  the  task  was  automated  was  varied  (0%,  20%,  40%,  60%  and  100%) 
in  addition  to  6  LOAs  selected  as  representative  based  on  the  earlier  study 
(Manual,  Batch  Processing,  Shared  Control,  Blended  Decision  Making, 
Supervisory  Control  and  Full  automation).  In  addition,  a  secondary  gage 
monitoring  task  was  employed  as  previous  studies  have  indicated  that 
automation  complacency  may  only  occur  when  the  operator  is  dual  tasked.  The 
earlier  LOA  study  found  that  SA  was  surprisingly  high  under  high  levels  of 
automation  (contrary  to  previous  findings),  possibly  due  to  the  fact  that  subjects 
had  not  other  task  to  perform  other  than  monitoring  the  automation.  Therefore, 
a  secondary  task  was  added  to  test  this  hypothesis  and  see  if  SA  is  indeed 
negatively  affected  by  automation  mostly  under  vigilance  and  dual-task 
conditions,  both  factors  which  are  characteristic  of  tasks  in  the  aviation  domain. 

The  results  of  this  study  generally  confirmed  the  findings  of  the  previous  study  in 
relation  to  the  effects  of  LOA  on  performance,  workload  and  situation 
awareness.  The  insertion  of  the  secondary  guage  monitoring  task  did  not 
substantially  change  the  results  of  the  study  across  LOA;  performance  on  this 
task  mapped  to  the  ratings  on  the  subjective  workload  measure  indicating  it 
reflected  a  measure  of  cognitive  resources  available  in  each  condition  as  would 
secondary  task  workload  measure.  Situation  awareness  was  best  under 
manual  control,  full  automation,  and  shared  control. 

While  AA  partioning  strategies  had  a  significant  effect  on  subject  performance 
with  performance  improving  as  percentage  of  time  the  task  was  automated,  LOA 
was  a  much  stronger  driving  factor  in  affecting  human  performance  in  this  task. 
The  AA  strategy  used,  however,  was  the  primary  factor  driving  subjective 
workload  and  workload  as  measured  by  the  secondary  guage  monitoring  task. 
AA  strategey  did  not  significantly  affect  operator  situation  awareness. 

The  results  of  this  study  act  to  both  confirm  the  results  of  the  previous  study  and 
to  expand  upon  them  based  on  more  realistic  dual-task  and  vigilance 
conditions  and  to  examine  the  impact  of  AA  strategies.  These  results  are 
particularly  important  for  the  design  of  automated  systems  in  a  wide  variety  of 
aviation  applications  including  cockpits  and  air  traffic  control. 


(5)  Investigation  of  cue  characteristics  related  to  errors  in  situation  awareness 
comprehension 

Situation  awareness  (SA)  errors  have  been  linked  to  a  large  majority  of 
accidents  and  incidents  involving  human  error  in  aviation.  In  some  poor  SA  can 
lead  to  accidents,  loss  of  life,  and/or  poor  mission  performance.  In  other  cases 
poor  SA  can  be  recovered  from  prior  to  a  major  performance  error  occurring.  A 
significant  question  exists  as  to  how  does  an  operator  may  realize  his  or  her  SA 
is  off-the-mark,  allowing  for  activities  to  regain  SA,  while  in  some  cases  the 
person  may  continue  with  poor  SA  oblivious  to  information  that  should  point  out 
his  or  her  erroneous  situation  model. 

Two  types  of  events  may  occur.  When  some  piece  of  data  is  perceived  it  may 
be  misinterpreted  and  integrated  with  the  existing  mental  model  to  yield  a 
misrepresentation  of  the  situation.  In  other  cases,  it  may  indicate  that  the 
operator’s  mental  model  is  in  error  and  a  new  mental  model  is  selected 
allowing  the  operator’s  SA  to  be  in  line  with  the  actual  situation.  An 
inappropriate  choice  can  easily  sabotage  SA  efforts  for  quite  some  time.  It  is 
hypothesized  that  different  types  of  cues  may  be  more  likely  to  call  to  the 
operator’s  attention  that  they  have  inaccurate  SA.  Schema  bizarre  cues  are 
expected  to  be  more  likely  to  be  recognized  that  schema  irrelevant  cues. 

The  presence  of  unexpected  cues  (per  the  mental  model)  is  also  expected  to 
be  more  likely  to  be  noticed  than  the  absence  of  expected  cues. 

An  experimental  study  was  conducted  for  testing  these  hypotheses  within 
the  context  of  a  high  fidelity  air  traffic  control  simulation.  It  engaged  12 
subjects  who  were  all  experienced  controllers  in  ATC  traffic  scenarios  which 
included  three  types  of  errors  that  would  induce  an  incorrect  mental  model: 
incorrect  aircraft  ID  on  flight  strip,  incorrect  flight  plan  on  flight  strip,  and  pilot 
communication  error.  Cues  were  then  introduced  according  to  the  above 
taxonomy  (schema  bizarre,  schema  irrelevant,  schema  unexpected,  and 
absense  of  schema  expected)  that  should  indicate  to  the  controller  that  an 
error  had  been  made  that  needed  correction.  (All  scenarios  and  cues  were 
developed  through  a  subject  matter  expert  to  ensure  fidelity  and  accuracy.) 

The  study  found  that  schema  bizarre  cues  were  detected  and  the  error 
discovered  more  frequently  than  schema  irrelevant  cues,  conforming  to  the 
hypothesis.  Schema  unexpected  cues  were  not  detected  more  frequently 
than  the  absense  of  expected  cues,  however.  In  investigating  the  many 
cases  where  subjects  did  not  discover  that  an  error  had  been  made  (65%  of 
all  cases),  it  appears  that  majority  involved  a  representational  error.  The 
subjects  detected  the  cue,  but  made  assumptions  to  explain  it  way  so  as  to 
be  consistent  with  the  orignal  (but  erroneous  mental  model).  Very  often 
these  explanations  were  quite  far  fetched  and  subjects  did  not  tend  to  check 
their  assumptions. 

This  study  represents  a  novel  attempt  to  address  methods  for  improving  SA 
based  on  schema  theory.  Its  results  should  be  applicable  to  a  wide  range  of 
systems  in  which  people  rely  on  SA  for  performance,  including  aircraft  and  air 


traffic  control.  The  study  revealed  significant  difficulties  in  correcting  incorrect 
mental  models  that  underly  SA,  but  that  schema  bizarre  cues  were  the  most 
likely  to  result  in  a  correction  being  made. 
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